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Abstract 

This  paper  describes  a  massive  survey  of  Network  Hme  Protocol  (NTP)  servers  and  clients  in  order  to 
discover  statistics  of  time  and  frequency  errors,  as  well  as  determine  the  health  and  welfare  of  the  NTP 
synchronization  subnet  operating  in  the  global  Internet.  Among  the  conclusions  of  the  survey  are  that 
most  NTP  clocks  are  within  21  ms  of  their  synchronization  sources  and  all  are  within  29  ms  on  average. 
Howeva,  additional  errors  up  to  one-half  the  roundtrip  delay  are  possible,  but  relatively  rare.  There  is, 
however,  a  disturbing  incidence  of  inqtroperly  operating  servers  and  misguided  server  configurations. 
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1.  introduction 

Timekeeping  via  the  Internet  has  become  a  ubiquitous 
service  extending  to  well  over  100,000  public  servers 
and  clients  in  many  countries  of  the  world.  In  addition, 
an  uncounted  number  of  private  servers  and  clients  lurk 
behind  the  firewalls  of  many  large  government  and  cor¬ 
porate  networks.  There  have  been  several  previous  stud¬ 
ies  [1],  [3],  [4]  which  have  conqjiled  population  and 
error  statistics  using  time-dissemination  protocols  such 
as  the  Network  Time  Protocol  (NTP)  and  others.  How¬ 
ever,  only  [4]  considered  reaching  out  to  aU  hosts  and 
routers  in  the  global  Internet.  That  study  indexed  the 
Domain  Name  System  database  of  eight  years  ago  and 
found  about  21,000  hosts  that  returned  time  in  one  fash¬ 
ion  or  another.  The  newer  studies  used  only  a  small  pop¬ 
ulation  to  evaluate  protocol  and  algorithm 
improvements  over  the  years  since  then. 

Since  the  present  day  Internet  has  e^qieiienced  an  explo¬ 
sion  of  numbers,  it  is  no  longer  acceptable  or  even  possi¬ 
ble  to  survey  Internet  hosts  and  routers  by  indexing  a 
public  directory  service.  However,  it  is  possible  to  dis¬ 
cover  them  for  an  organized  subnetwork  such  as  used  by 
NTP.  This  paper  presents  a  conqnehensive  survey  of  the 
NTP  servers  operating  in  the  global  Internet  of  today. 
While  the  search  engine  used  in  the  survey  could  find 
only  a  fraction  of  all  NTP  outposts  on  the  public  Inter¬ 


net,  it  did  find  182,538  network  paths  used  by  36,000 
servers,  231  of  these  synchronized  to  UTC  via  radio, 
satellite  or  modem.  Uncounted  others,  including  large 
numbers  of  personal  workstations  operating  as  broad¬ 
cast  clients,  escaped  detection.  The  survey  measured  the 
time  offsets  of  each  client  relative  to  its  server,  as  well  as 
related  variables  important  to  the  health  and  welfare  of 
the  NTP  subnet  itself. 

A  comprehensive  description  of  the  NTP  architecture, 
protocol  and  algorithms  is  beyond  the  scope  of  this 
paper,  but  can  be  found  in  [2]  and  citations  found  there. 
The  NTP  timekeeping  network  consists  of  a  hierarchical 
tree  of  primary  (stratum  1)  time  servers  at  the  root. 
These  provide  synchronization  to  secondary  servers  at 
increasing  stratum  levels  (hops)  to  the  leaves  of  the  tree. 
Figure  1  shows  the  functional  organization  of  a  NTP 
server.  The  server  sends  a  message  to  one  or  more  other 
peer  servers  and  expects  replies  from  each  of  them  at 
some  later  time.  From  the  four  timestanqts  obtained 
upon  departure  and  arrival  of  the  request  and  response 
messages,  the  server  calculates  the  clock  offset  and 
roundtrip  delay  relative  to  each  peer.  The  clock  filter 
algorithm  selects  the  best  measurements  from  each  peer 
and  the  intersection  and  clustering  algorithms  the  best 
combination  of  several  peers  operating  simultaneously. 
The  actual  local  clock  oscillator  (LCO)  time  and  fre¬ 
quency  are  disciplined  using  an  adaptive  parameter, 
hybrid  phase/frequency-lock  loop. 
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Figure  1.  NTP  Server  Functional  Organization 


2.  Survey  Methodology  The  data  for  each  server  consist  of  a  spreadsheet,  where 

each  line  of  the  spreadsheet  corresponds  to  a  protocol 
The  survey  which  generated  the  bulk  of  the  results  pre-  instantiation,  called  an  association  between  the  server 
sented  in  this  paper  was  performed  over  some  months  in  and  its  peer  server,  as  shown  in  Figure  1.  The  peer  is 

late  1995.  The  goal  was  to  find  as  many  NTP  servers  either  another  server  or  a  primary  reference  clock,  usu- 

operating  in  the  Internet  as  practical  and  determine  their  ally  a  GPS  receiver.  Typical  server  configurations 

performance  with  respect  to  time  and  frequency  errors,  include  several  associations,  which  run  simultaneously 
as  well  as  the  robusmess  and  stability  of  the  NTP  subnet  in  order  to  provide  redundancy  and  diversity.  The  asso- 

itself.  The  results  serve  as  a  baseline  calibration  of  the  ciation  data  include  the  endpoint  Internet  addresses, 

subnet  performance,  as  well  as  useful  insights  into  sub-  stratum,  mode,  reachability  status,  relative  clock  offset, 
net  configuration  and  security  management  strategies.  roundtrip  delay  and  error  estimate  called  dispersion. 


The  search  engine  used  a  variety  of  techniques,  ranging 
from  indexing  available  public  databases  to  exploring 
the  Internet  NTP  subnet  in  real  time  with  available  mon¬ 
itoring  protocols  and  tools.  In  the  following,  subnet 
without  the  qualifier  NTP  means  any  subtree  of  the  glo¬ 
bal  NTP  subnet  tree.  The  strategy  was  to  identify  as 
many  servers  as  possible  from  public  lists,  such  as  the 
database  of  public  servers  maintained  at  the  NTP  web 
page  www.eecis.udel.edu/~ntp.  There  are  currently  69 
public  primary  (stratum  1)  servers  and  105  public  sec¬ 
ondary  (stratum  2)  servers  listed  in  that  database.  In  the 
initial  round,  these  servers  were  queried  using  NTP 
monitoring  and  control  protocols  to  identify  their  client 
servers.  In  the  next  round,  those  servers  were  queried  to 
identify  their  client  servers  and  so  on.  The  survey  con¬ 
tinued  for  many  rounds  until  no  new  servers  could  be 
found.  Occasionally,  the  identity  of  a  new  server  was 
discovered  by  other  means  and  the  survey  continued  for 
new  rounds  as  necessary. 

It  was  vital  in  the  planning  and  execution  of  the  survey 
to  avoid  network  and  server  congestion  due  to  the  query 
messages  themselves.  Therefore,  the  intervals  between 
messages  were  randomized  over  moderately  long  inter¬ 
vals  in  the  order  of  minutes.  Due  to  the  volume  of  data 
involved,  individual  rounds  required  up  to  days  to  com¬ 
plete  and  the  entire  survey  required  many  rounds. 


Each  association  operates  in  a  designated  mode, 
depending  on  whether  the  server  can  synchronize  the 
peer  or  whether  the  peer  can  synchronize  the  server. 
Symmetric  modes  are  used  when  either  peer  can  syn¬ 
chronize  the  other  and  where  mutual  backup  is  neces¬ 
sary  in  multiple-peer  subnets.  Client/server  modes  are 
used  in  subnets  of  modest  size  and  where  the  highest 
accuracy  is  required.  Broadcast  modes  are  used  in  large 
subnets  where  the  delays  are  small  and  the  accuracy 
requirements  not  demanding. 

The  data  analysis  phase  of  the  project  considered 
230,774  associations,  but  discarded  818  with  misconfig- 
ured  servers,  3,673  with  broadcast  servers,  25,640  with 
unreachable  servers,  17,195  with  invalid  stratum  and 
1,293  duplicates,  leaving  180,520  representing  valid 
measurements  of  clock  offset  and  roundtrip  delay.  In 
these,  36,478  distinct  Internet  addresses  were  found, 
each  identifying  a  NTP  server.  It  should  be  noted  that 
these  servers  represent  only  a  fraction  of  all  servers  in 
the  NTP  subnet.  Clients  operating  in  broadcast  mode 
were  not  found,  nor  were  many  others  on  private  net¬ 
works  or  behind  firewalls.  Since  these  statistics  were 
determined,  a  few  additional  valid  associations  were 
found,  bringing  the  total  to  182,538  associations  and 
38,722  servers. 
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Figure  2.  Association  Population  by  Stratum 


In  the  above  totals,  there  were  3,673  servers  operating 
in  broadcast  modes,  or  about  one  in  ten.  Tbe  number  of 
clients  of  these  servers  is  unknown,  since  broadcast  cli¬ 
ents  ordinarily  do  not  exchange  messages  with  their 
servers.  In  fact,  most  broadcast  servers  have  tens  to 
hundreds  of  cUoits,  so  the  actual  client  population  is 
considerably  underestimated.  The  1,293  duplicates  rep¬ 
resent  symmetric  associations,  where  each  of  two  peers 
agree  to  back  up  the  other.  Ordinarily,  the  offsets  mea¬ 
sured  by  symmetric  peers  have  nearly  the  same  magni¬ 
tude,  but  opposite  sign,  and  the  delays  are  nearly  the 
same,  but  this  was  not  a  factor  in  the  analysis. 

Figure  2  shows  the  number  of  associations  by  stratum. 
Among  the  182,538  associations,  there  are  13,880  dif¬ 
ferent  servers  with  an  overall  average  of  13.2  associa¬ 
tions  per  server.  The  maximum  number  of  associations 
per  server  is  652,  but  this  understates  the  true  value, 
since  the  TnaximnTn  numb^  the  server  monitoring 
cache  can  hold  is  near  this  number.  From  other  data  dis¬ 
cussed  later,  higher  values  to  734  have  been  observed 
on  one  server  which  is  certainly  not  the  most  heavily 
loaded.  The  fact  that  at  most  stratum  levels  the  maxi¬ 
mum  and  the  top  ten  have  about  the  same  number  of 
associations  and  the  overall  average  is  much  Iowa  sug¬ 
gests  the  population  is  dominated  by  a  relatively  small 
numba  of  heavily  loaded  servas  and  the  remainder  rel¬ 
atively  lightly  loaded.  About  half  the  numba  of  servers 
opaate  at  stratum  3,  which  usually  function  as  d^art- 
ment  servas  dependent  on  redundant  campus  servers 
opaating  at  stratum  2.  The  servas  shown  opaatmg  at 
stratum  6  and  above  are  very  likely  broken  a  unstable. 

3.  Performance  Indicators 

The  three  most  significant  factors  that  affect  timekeep¬ 
ing  accuracy  are  network  jitta,  oscillator  frequency  sta¬ 
bility  and  asymmetric  network  delays.  The  influence  of 
these  three  factors  varies  widely  in  the  NTP  subnet, 
depending  on  the  particular  network  path  and  particular 
oscillator  implementation.  The  survey  results  allow 
each  of  these  factas  to  be  studied,  evaluated  and  dis¬ 


played  as  graphs  and  tables  and  statistics  doived  from 
them.  Howeva,  the  survey  was  conducted  ova  a  period 
of  months,  whae  the  particular  network  conditions 
wae  not  compiaely  stationary.  For  this  reason,  the 
results  diould  be  regarded  as  an  approximation  to  a  true 
sn^sbot. 

3.1  Time  Offsets 

The  most  important  indicator  of  timekeeping  pafor- 
mance  is  the  time  offset  measured  by  the  serva  relative 
to  the  pea  serva  for  each  association.  Figure  3  shows 
the  cumulative  distribution  function  (CDF)  of  time  off¬ 
sets  for  38,722  servas,  whae  the  median  is  23.3  ms, 
mean  234  ras  and  maximum  686  ms.  Howeva,  3,833 
servas  with  offsets  greata  than  128  ms  wae  not  syn¬ 
chronized  by  NTP  at  the  time  of  the  survey.  When  these 
servas  are  excluded,  the  median  is  20.1  ms  and  mean 
28.7  ms.  Note  that  these  results  represent  the  global 
population  at  large,  whaeas  clients  on  typical  LANs 
have  offsets  genaally  less  than  a  millisecond  [3]. 

According  to  Figure  2,  of  the  total  population  of  38,722 
servas,  220  primary  servas  at  the  root  of  the  NTP  sub¬ 
net  tree  provide  synchronization  to  13,660  secondary 
servas  and  these  provide  synchronization  to  the  24,842 
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Rgure  4.  Systematic  Frequency  Offsets  Figure  5.  Oscillator  Phase  Errors 

remaining  servers  at  the  leaves  of  the  tree,  hi  the  present  500  and  the  time  constant  varies  with  the  measure- 

NTP  design,  each  serva  on  the  path  from  the  leaves  to  ment  interval  from  64  s  to  1024  s. 

the  root  independently  disciplines  its  clock  relative  to 

the  servers  at  die  next  lower  stratum,  in  order  to  provide  Oscillator  phase  erroK  are  due  both  to  frequency  esti- 
a  stable,  low-jitter  reference  for  local  applications.  Thus,  errors  and  jitter  in  the  network.  Figure  5  shows  the 

the  figure  shows  the  offset  of  a  cUent  relative  to  its  special 

server,  not  the  offset  relative  to  the  primary  servers  at  the  ^®ss  131  outliers  with  errors  over  1  s,  from 

root.  In  principle,  the  protocol  could  determine  this  by  ^ 

accumulating  the  offsets  on  each  hop  along  the  path  to  ^  occur  only  if  the  peer  is  defective  and  the 

the  primary  servers.  However,  this  statistic  would  result  synchronized  to  another  peer.  Errors  over  128 

in  large  jitter  and  be  unsuitable  if  used  to  discipline  the  occur  only  during  initial  synchronization  after 

system  clock.  In  the  interest  of  an  economical  packet  reboot  or  infrequently  due  to  network  or  peer  server  dis- 

header,  it  was  not  included  in  the  NTP  protocol  design.  rap^ons.  If  these  servers  are  excluded  from  the  popula¬ 

tion,  the  median  is  8.0  ms  and  mean  16.7  ms. 


3,2  Clock  Discipline  Errors 

The  NTP  clock  discipline  algorithm  operates  to  mini¬ 
mize  both  phase  and  frequency  errors,  lb  the  extent  it 
makes  good  predictions  based  on  measured  time  offsets 
produced  by  the  protocol,  phase  errors  due  to  systematic 
frequency  offsets  can  be  minimized.  However,  in  order 
to  provide  millisecond  accuracies  wifli  measurement 
intervals  up  to  1024  s,  it  is  necessary  to  estimate  the  cur¬ 
rent  frequency  to  less  than  one  PPM,  even  when  the  sys¬ 
tematic  frequency  offset  is  hundreds  of  PPM.  The 
systematic  frequency  offsets  and  oscillator  phase  errors 
of  19,873  servers  were  detmnined  from  a  special  sur¬ 
vey.  Figure  4  shows  the  CDF  for  these  servers  less  593 
outliers  with  offsets  greater  than  500  PPM,  from  which 
the  median  is  38.6  PPM  and  mean  78.1  PPM.  Fietiuency 
offsets  greater  than  500  PPM  are  not  credible,  since  that 
is  the  limit  of  the  discipline  loop  c^ture  range. 


3.3  Asymmetric  Delays 

The  NTP  protocol  design  does  not  permit  an  absolute 
measurement  of  time  difference  between  severs,  unless 
an  external  common-view  means  is  available,  as  in  [1]. 
As  a  result,  clock  offset  errors  cannot  be  distinguished 
from  asymmetric  network  delays.  There  is  evidence  to 
suggest  that  errors  due  this  cause  are  not  uncommon; 
however,  by  design  the  errors  due  to  asymmetric  delays 
are  bounded  by  half  the  roundtrip  delay  [2],  although 
errors  this  large  are  rare.  Figure  6  shows  the  CDF  for  the 
delays  measured  between  the  38,722  servers  used  in  the 
time  offset  survey.  The  results  show  median  delay  118 
ms,  mean  186  ms  and  rnaxinimn  a  whopping  1.9  s.  As 
with  oQier  survey  data  analyzed,  the  tails  on  the  distribu¬ 
tions  can  be  very  long.  Ffom  anecdotal  evidence,  the 
larger  values  due  to  the  widely  varying  incidence  of  con¬ 
gestion  in  the  Internet  of  modem  times. 


While  most  clock  oscillators  are  moderately  well 
behaved,  there  is  a  significant  tail  on  the  distribution. 
The  results  demonstrate  that  the  NTP  clock  discipline 
must  have  a  very  large  capture  range  and  an  adaptive 
time  constant.  In  the  present  design,  the  capture  range  is 


From  anecdotal  evidence  and  previously  reported  results 
[1],  [3],  the  delay  asymmetry  is  seldom  more  than  one- 
tenth  the  roundtrip  delay.  For  LANs  with  delays  in  the 
low  milliseconds,  the  asymmetric  errors  are  generally 
negligible.  Even  on  long  paths  spanning  the  Atlantic 
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Rgure  7.  Data  Mitigation 


where  propagation  delays  dominate,  the  errors  are  small 
compared  to  those  due  to  network  jitter.  However,  in 
cases  where  one  direction  uses  cable  and  the  other  satel¬ 
lite,  for  example,  or  where  two  peers  are  connected  to 
different  providers  with  very  different  network  grids,  the 
errors  can  be  substantial. 

3.4  Data  Mitigation 

A  good  idea  of  the  general  health  of  the  NTP  subnet  can 
be  determined  from  the  status  code  maintained  for  each 
association.  Figure  7  shows  a  breakdown  of  the  182,538 
associations  considered  previously.  The  table  is  best 
interpreted  starting  from  the  bottom.  At  each  step 
upward  from  the  bottom,  the  indicated  number  of  asso¬ 
ciations  are  discarded  for  the  reason  given  on  the  same 
Une.  The  line  under  the  heading  line  shows  the  number 
of  associations  actually  used  to  discipline  the  system 
clock.  Since  there  is  only  one  such  association  per 
server,  this  number  represents  the  actual  number  of  serv¬ 
ers  synchronized  by  NTP.  Since  there  were  38,722  serv¬ 
ers  resulting  from  the  time  offset  survey,  the  remaining 
4,338  servers  may  be  unsychronized,  about  11  percent 
of  the  total.  Caution  is  advised  when  making  such  con- 
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elusions,  since  the  survey  errors  due  to  changing  net¬ 
work  conditions  could  well  account  for  the  differences. 

For  the  remaining  lines  of  Figure  7,  a  few  notes  will 
clarify  the  operations  involved.  The  inta-section  algo¬ 
rithm  detects  and  discards  associations  which  show  a 
time  interval  outside  the  correct  time  interval  deter¬ 
mined  for  the  majority  of  associations.  Standby  associa¬ 
tions  are  those  deemed  correct,  but  not  among  the  lowest 
ten  in  order  of  increasing  synchronization  distance. 
When  more  than  ten  associations  are  admitted  to  the 
clustering  algorithm,  excessive  clockhopping  can  occur 
among  the  peers,  which  can  result  in  excessive  system 
clock  jitter.  The  relatively  high  number  of  standby  asso¬ 
ciations  was  traced  to  a  provido'  using  a  very  misguided 
configuration  where  each  of  about  100  NTP  servers 
maintained  associations  with  all  of  the  others. 

The  clustering  algorithm  repeatedly  discards  outlier 
associations  until  either  a  miniTniiTn  number  of  three 
remain  or  the  residual  saiiq>le  variance  cannot  be 
reduced  by  further  discards.  The  remaining  population, 
including  the  tep  three  lines  in  the  table,  represent  can¬ 
didate  associations  considered  the  pick  of  the  litter. 
Selection  among  the  candidates  to  determine  the  ones 
actually  used  to  discipline  the  system  clock  is  deter¬ 
mined  on  the  basis  of  synchronization  distance.  The 
combining  algorithm  computes  the  offset  provided  to 
the  clock  discipline  as  a  average  of  the  available  candi¬ 
dates  weighted  by  this  distance.  Thus,  the  average  num¬ 
ber  of  associations  used  to  synchronize  the  system  clock 
is  1.75. 

3.5  Primary  Servers 

The  accuracy  and  stability  of  the  global  NTP  subnet 
depends  on  the  refCTence  clocks  used  to  synchronize  the 
primary  servers.  There  are  many  different  means  used  to 
do  this,  including  radio,  satellite,  telephone  modem  and 
cesium  clock  ensemble.  Figure  8  shows  the  population 
characteristics  of  the  231  primary  reference  clocks 
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found  in  the  survey.  Since  this  survey  was  completed, 
the  national  time  dissemination  services  of  France, 
Chile  and  Italy  have  become  available  and  additional 
time  servers  operated  by  NIST  and  USNO  have  come 
online,  making  the  total  operating  today  significantly 
larger.  The  surprisingly  large  number  of  DCF77  refer¬ 
ence  clocks  is  undoubtedly  due  to  the  very  inexpensive 
receivers  available  for  this  signal  in  Europe.  In  the  inter¬ 
val  since  this  survey  was  completed,  and  as  inexpensive 
GPS  receivers  have  come  on  the  market,  new  servers  arc 
using  GPS. 

In  addition  to  the  231  primary  servers  found,  some  88 
servers  were  found  where  the  reference  clock  had  either 
failed  or  had  been  misconfigured.  In  the  NTP  design, 
reference  clocks  are  treated  as  ordinary  peer  servers,  so 
many  primary  servers  maintain  associations  for  other 
servers  as  backup.  When  the  radio  fails,  the  other  associ¬ 
ations  kick  in  and  the  server  continues  to  operate  at  a 
higher  stratum. 

In  some  configurations,  a  special  reference  clock  driver 
provides  backup  when  all  other  sources  fail.  The  driver 
simulates  a  reference  clock  operating  at  some  higher 
stratum,  usually  3.  If  all  sources  of  lower  stratum  fail, 
the  free-running  server  system  clock  becomes  the  NTP 
reference  and  the  clients  depending  on  the  server  syn¬ 
chronize  to  it  At  the  time  of  the  survey,  1,502  servers 
were  found  to  use  this  feature. 

4.  A  Day  in  the  Life:  of  a  Busy  NTP  Server 

While  some  NTP  servers  are  dedicated  timekeepers, 
most  serve  other  functions  as  well,  including  those  of 
routers,  file  servers  and  ordinary  workstations.  It  is  of 
some  concern  to  determine  how  much  the  resource 


requirements  of  the  timekeeping  function  impact  on 
these  other  functions.  Of  principal  concern  are  the  pro¬ 
cessing  and  network  overhead  resulting  from  client 
requests  to  a  busy  time  server. 

Timekeeper  rackety,  one  of  the  most  experienced  time¬ 
keepers  in  the  business,  is  a  well  known  primary  time 
server  with  clients  all  over  the  world.  The  configuration 
of  this  server  in  the  context  of  the  other  timekeeping 
gear  in  our  laboratory  is  shown  in  Figure  9.  Together 
with  other  timekeepers  pogo  and  bamstable,  this  Sun 
Microsystems  SPARCstation  IPC  functions  as  a  file 
server,  print  server  and  router  as  well,  although  in  this 
machine  these  functions  have  low  demand.  An  Austron 
2201A  GPS  receiver  functions  as  the  primary  reference 
clock  and  a  Spectracom  WWVB  receiver  as  backup. 
The  operating  system  kernel  is  modified  to  use  the 
pulse-per-second  (PPS>  signal  from  a  Hewlett-Packard 
5061 A  cesium  oscillator  to  discipline  the  system  clock, 
generally  within  a  few  tens  of  microseconds.  While 
implemented  expressly  for  the  SunOS  operating  system, 
these  kernel  modifications  are  also  in  current  Digital 
Unix  and  Sun  Solaris  operating  systems. 

In  order  to  calibrate  the  resource  demands  of  the  time¬ 
keeping  function,  ordinary  Unix  commands  were  used 
to  measure  the  CPU  time  consumed  by  the  NTP  daemon 
over  about  a  day,  which  came  to  1.54  percent.  A  net¬ 
work  sniffer  program  captured  all  of  the  NTP  packets 
for  rackety  that  flew  the  DCnet  wires  during  that  day 
and  saved  them  for  later  analysis.  The  734  clients  sent 
an  average  of  6.4  packets  per  second,  which  corresponds 
to  a  mean  interarrival  interval  of  157  s  for  each  client. 
This  is  curious,  since  after  an  initial  training  period, 
most  NTP  Version  3  clients  should  increase  the  poll 
interval  in  steps  to  1024  s.  The  fact  the  interarrival  inter- 


370 


val  is  so  small  suggests  there  are  stiU  a  large  number  of 
previous  versions  in  use. 

Each  input  packet  generated  on  average  0.64  ou^ut 
packets  and  required  a  total  of  2.4  ms  of  CPU  time  for 
the  request/response  transaction.  It  is  not  known  how 
many  of  the  734  clients  were  using  cryptographic 
authentication;  but,  if  so,  the  time  to  generate  the  request 
and  response  cryptosums  on  an  IPC  is  about  300  p.s,  but 
only  about  30  p.s  on  an  UltraSPARC  170. 

Rackety  is  connected  via  a  10-Mb/s  Ethernet,  routers 
and  1.544  Mb/s  T1  circuits  to  Internet  backbone  net¬ 
works  as  shown  in  Figure  9,  While  the  T1  circuits  are 
not  dedicated  to  NTP  traffic,  the  other  customers  are 
few.  Averaged  over  the  day,  the  76-octet  NTP  packet 
rate  was  measured  at  10.5  packets  per  second.  Exclusive 
of  overhead  bits,  this  is  .064  percent  of  the  Ethernet 
capacity  and  a  maximum  of  0.21  percent  on  each  direc¬ 
tion  of  the  T1  capacity.  The  conclusion  to  be  drawn  here 
is  that  the  resources  required  for  NTP  are  minimal,  even 
when  hundreds  of  clients  are  involved  and  even  with  a 
slow  processor. 

5.  Conclusions 

The  results  of  the  survey  demonstrate  that  most  NTP 
clocks  are  within  21  ms  of  their  servers  and  all  are 
within  29  ms  on  average.  The  oscillator  phase  error  may 
add  a  few  milliseconds  to  these  figures,  but  in  most 
cases  can  be  neglected.  There  may  be  additional  errors 
due  to  asymmetrical  paths,  but  this  is  bounded  by  one- 
half  the  roundtrip  delay.  While  it  is  not  possible  using 
the  NTP  protocol  itself  to  estimate  the  errors  when  mul¬ 
tiple-hop  paths  are  involved,  a  reasonable  estimate  may 
be  the  number  of  hops  times  these  figures.  It  is  clear 
from  the  graphs  that  the  error  distributions  can  have  sur¬ 
prisingly  large  tails  well  above  these  figures,  in  some 
cases  due  to  network  congestion  and  in  other  cases  due 
to  improperly  operating  servers  or  misguided  choice  of 
servers. 


The  roundtrip  delay  statistic  has  proven  very  useful,  not 
only  to  assess  timekeeping  performance,  but  in  the  gen¬ 
eral  study  of  the  worldwide  Internet  paTormance.  The 
fact  that  NTP  servers  determine  this  statistic  on  a  regular 
basis  suggests  the  use  of  NTP  as  a  network  monitor  with 
a  coupling  to  the  monitoring  infrastructure  using  the 
Simple  Network  MonitOTing  Protocol  (SNMP).  In  fact, 
this  has  been  done  on  an  experimental  basis  and 
described  in  a  report  now  in  preparation. 

Perhaps  the  most  disturbing  results  of  the  survey  is  the 
significant  fi-action  of  servas  showing  improper  operat¬ 
ing  parameters  or  incorrectly  configured  servers.  In 
many  cases,  this  is  due  to  benign  neglect  on  the  part  of 
the  operator,  since  timekeeping  is  not  usually  regarded 
as  a  revenue  service,  and  providing  NTP  service  to  the 
community  is  a  volunteer  sport.  This  problem  has  been 
addressed  in  the  design  of  the  next  version  of  the  NTP 
protocol  in  the  form  of  features  to  discover  servers  and 
configure  the  NTP  subnet  automatically  and  without  the 
chance  of  operator  error. 
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Questions  and  Answers 


BILL  KLEPCZYNSKI  (INNOVATIVE  SOLUTIONS  INTERNATIONAL):  You  use  the  terni  “Stratum 
1”.  Stratum  1  to  the  telecommunications  people  means  a  microsecond  and  a  part  of  10  to  the  llth  in 
frequency.  Do  you  use  that  in  the  same  sense  or  not,  “Stratum  1”? 

DAVID  MILLS  (UNIVERSITY  OF  DELAWARE):  Of  course  not.  You  might  say  if  we  are  talking  about 
Stratum  1  being  a  part  in  10  to  the  12^^^  we  are  perhaps  about  nine  orders  of  magnitude  worse  than  that.  I 
had  not  intended  to  poach  on  the  nomenclature  for  the  telecommunications  industry,  but  I  had  to  have 
something.  So  “Stratum  1”  it  is. 
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